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ABSTRACT: The development of the constraint-referenced analytics tool for monitoring algebra 
learning activities presented here came from the desire, firstly, to take a more quantitative look 
at student responses in collaborative algebra activities, and secondly, to situate those activities in 
a more traditional introductory algebra setting focusing on procedural understanding. Procedural 
skill was analyzed by modelling the complexity of attempts to make equivalent transformations 
of algebraic expressions. The constraint-referenced analytics system uses log files of student 
inputs on a classroom network of handheld devices to measure success rate as students attempt 
to replace one algebraic expression with another equivalent expression. The analytics engine 
produced psychometrically verifiable results. Moving averages of student performance revealed 
that when students experienced a period of struggle and persisted in attempting similar 
transformations, an apparent conceptual shift led to subsequent success. Students also 
responded to periods of struggle by switching to familiar tasks or choosing non-participation. 


Keywords: Learning analytics, measurement, psychometrics, assessment, algebra learning, 
procedural knowledge, conceptual knowledge, collaborative networks, classroom networks. 

1 INTRODUCTION 

The recent emergence of tools and techniques for educational data mining and learning analytics have 
introduced an array of new resources for automatically monitoring, assessing, and supporting student 
performance within computer-based learning environments (Baker & Inventado, 2014). Automated 
approaches have been used to capture a wide variety of skills and behaviours, including science inquiry 
skills in online microworlds (Gobert, Sao Pedro, Raziuddin, & Baker, 2013), player learning within video 
game play (Halverson & Owen, 2014), help-seeking behaviours in online problem-solving environments 
(Roll, Baker, Aleven, & Koedinger, 2014), and programming fluency in computer science education 
contexts (Blikstein et al., 2014). To date, many of these efforts have focused on online environments, 
but the potential for analytics tools and approaches to support face-to-face instruction in classroom 
settings may be equally rich. 

This article investigates one such approach, implemented in the context of classroom network tools 
developed for an introductory high school Algebra course. We present an initial exploration of 
constraint-referenced analytics, an automated way to measure the complexity of processes involved in 
making equivalent algebraic transformations by referencing digitally encoded constraint-based models. 
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The design of the analytics engine combines constraint-based models of algebra found in intelligent 
tutors (Mitrovic, Mayo, Suraweera, & Martin, 2001) with models of complexity informed by algebra¬ 
learning literature to develop an automated form of measurement capable of detecting periods of 
struggle and student responses to that struggle. Measurements were taken by parsing log file records of 
student inputs on a classroom network of handheld devices while they worked in pairs to construct or 
transform algebraic expressions and equations. The approach produced psychometrically verifiable 
results without embedding preconceived assessment items or performance measurements in the design 
of the learning activities. 


Constraint-based models for intelligent tutors emerged from the need to encode contextual properties 
rather than rules-based procedures for ill-defined and unstructured domains of knowledge (Mitrovic, 
2012). Whereas rules-based models predefine a set of outcome goals and correct students when they 
stray from a productive path, constraint-based models define what constitutes valid activity within the 
domain, simply explaining the reasons for a violation of the constraints. For example, a student given 
the equation 2(x + 6) = 3x(x + 6) may distribute the terms producing 2x + 12 = 3x 2 + 18x. A rules-based 
system focused on solving for x would prompt the user to try again. A constraint-based system would 
allow the transformation without comment since the procedure did not violate any algebraic domain 
constraints. If the student mistakenly attempted to distribute the right side of the equation as 3x 2 + 6, 
the constraint-based system would explain that both resulting terms must contain at least an x term 
since the distributed term contains an x. A constraint-based system does not trace progress through a 
predefined set of solution paths, but captures violations of constraints that define a particular context 
without reference to a specific goal. Students in traditional algebra classes often encounter tasks that 
involve replacing one expression with an equivalent expression (Choppin, Clancy, & Koch, 2012). The 
relative complexities of the algebraic context and the transformation attempt, and patterns of repeated 
attempts, may reveal periods of struggle and distinct ways that students respond to that struggle. 
Below, we present an exploratory investigation of the ways a constraint-based approach to analyzing 
these transformations might be used to monitor student learning in Algebra. 

2 THEORETICAL FRAMEWORK 


In this section, we frame the present study with a definition of conceptual knowledge from a brief 
review of definitions found in mathematics literature, followed by a review of sources of the difficulties 
that students face when learning algebra. The section concludes with a presentation of the rationale for 
constraint-referenced measurement and its intelligent tutor roots. 

2.1 Conceptual Knowledge in Mathematics 

Mathematical thinking is generally taken to comprise both procedural and conceptual knowledge 
(Hiebert, 2013). While researchers generally agree on both the definition and measurement of 
procedural knowledge as the ability to perform mathematical processes to reach a desired outcome, 
less agreement exists for conceptual knowledge. In an exhaustive review of mathematics education and 
educational psychology literature, Crooks and Alibali (2014) identified six definitions of conceptual 
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knowledge: 1) connection knowledge, 2) general principle knowledge, 3) knowledge of principles 
underlying procedures, 4) category knowledge, 5) symbol knowledge, and 6) domain structure 
knowledge. Researchers also made a distinction between explicit conceptual knowledge that could be 
expressed or verbalized and implicit knowledge that could be demonstrated but not explained. 
Conceptual understanding, defined as knowledge of principles underlying procedures, was primarily 
found in studies of understanding cardinality or inversion among preschool and elementary students. 
Less than one quarter of studies on equivalence contained any definition of conceptual knowledge, and 
those that did variously defined it as connection knowledge, general principle knowledge, category 
knowledge, or symbol knowledge. 


2.2 Sources of Difficulty for Algebra Learners 

Mathematics education researchers have documented several sources of difficulty that students face 
when making the transition from arithmetic to algebra. Here we summarize three of these sources of 
difficulty: mathematical task complexity, symbolic complexity and meaning, and structural-procedural 
duality, reserving more space for a review of cognitive complexity. 


A significant portion of many traditional algebra courses involves reducing algebraic expressions to their 
simplest form. The number of procedural steps necessary and the kinds of processes required contribute 
to the mathematical complexity of the task. Researchers measure procedural skill by assessing the 
sophistication and appropriateness of procedural methods that learners use (Ball, Pierce, & Stacey, 
2003; Warren, 2003; Saldanha & Kieran, 2005; Rossi, 2008). Learners sometimes invent their own 
sequential procedures in both algebra and arithmetic when they encounter unfamiliar contexts 
(Linchevski & Livneh, 1999; McNeil & Alibali, 2005; Ambrose, Baek, & Carpenter, 2013). 

Familiar mathematical symbols that have very few meanings in arithmetic take on multiple meanings 
depending upon the context in algebra. Studies found that algebra learners used surface features, visual 
cues, and symbolic meaning to make operational decisions (Kirshner, 1993). The symbolic context 
influenced the words learners used and the meaning they attached to the symbols (Wagner & Parker, 
1993). Linchevski and Livneh (1999) found that student misconceptions of the meaning of mathematical 
symbols were consistently applied in different algebraic situations, which led to incorrect answers in 
some contexts and correct answers in others. These misconceptions were traced to operational 
processes learned in arithmetic (Booth, 1988; Linchevski & Livneh, 1999). 


Mathematics by nature is both structural and operational, and many sources of difficulty identified by 
researchers involve this dual nature of mathematical concepts (Sfard, 1991; Sfard & Linchevski, 1994). 
The notion of equivalence gives algebra much of its power, allowing any algebraic expression to be 
replaced by any equal expression (Gattegno, 1974; Kieran, 1981). A structural knowledge of equality as a 
relation has been associated with success in algebra (Denmark et al., 1976; Jones, Inglis, Gilmore, & 
Dowens, 2012; Jones, Inglis, Gilmore, & Evans, 2013). Structural and operational knowledge in algebra 
literature is analogous to conceptual and procedural knowledge previously reviewed in other 
mathematics education research. 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


145 


JOURNAL OF LEARNING ANALYTICS 


S8LAR 


(2016). Constraint-referenced analytics of algebra learning. Journal of Learning Analytics, 3(3), 143-169. 
http://dx.doi.Org/10.18608/jla.2016.33.8 

One particular study brought the connection between conceptual and procedural knowledge into focus. 
Filloy and Rojano (1989) found that 13- and 14-year-old students could use inverse arithmetic 
operations to solve equations such as 3x + 2 = 6, but resorted to a guess and check approach based on 
arithmetic when confronted with 5x + 2 = 2x + 6. In no instance did students spontaneously operate on 
the unknown in the absence of instruction. A structural conception of the unknown as an object is 
needed for operations to be performed with it or on it. Results were based upon a measure of 
procedural skill as a reflection of conceptual understanding of the principles underlying the procedures. 
Developing the ability to operate with and on the unknown has been established as an important aspect 
of pre-algebra learning (Linchevski, 1995). 


2.3 Cognitive Complexity 

Cognitive complexity can be characterized as the number of cognitive processes that must be 
coordinated to accomplish a particular task. Research that compared expert and novice performance led 
to the notion of chunking, knowledge constructed by chunking together disparate patterns and concepts 
into meaningful wholes called schema. Experts were able chunk together larger, more complex 
schemata than novices (Chase & Simon, 1973; Chi, Feltovich, & Glaser, 1981). These studies show that, 
with enough effortful practice, novices can develop expertise in many areas of cognitive skill. 


Conceptual understanding of mathematics largely develops through recognizing patterns and relations 
between mathematical elements, and being able to classify them in different ways (Hiebert, 2013). 
Procedural knowledge cannot be reported, it must be performed (Rittle-Johnson, Siegler, & Alibali, 
2001), and researchers acknowledge that executing a series of procedural steps to achieve a particular 
answer can be done without much more than recognition of surface features. However, conceptual and 
procedural knowledge are now seen as inextricably intertwined. It is hard to imagine conceptual 
understanding without procedural skill. Rote procedural skill is more likely to occur when the student is 
taught the procedure than when they figure it out for themselves (Hiebert, 2013). Students are able to 
develop formal mathematical procedures better when they engage in sense making activities (Choppin 
et al., 2012) and when they are able to persist in problem solving (Renninger, Hidi, & Krapp, 2014). Initial 
failure can be productive if the students persist long enough and learn from their failures until they 
reach a moment of insight that leads to a conceptual shift in understanding. A class of problems that 
were once a struggle become easier to solve (Kapur, 2011; 2016). 

2.4 Intelligent Tutors and Algebra Learning 

Cognitive tutors focused on procedural understanding of mathematics (Mitrovic, Koedinger, & Martin, 
2003). Students learned how to get an answer or reach a predetermined goal-state, but struggled when 
making the transition to algebra. A structural understanding of mathematics that enabled students to 
see the relations between mathematical elements was found to be associated with success in algebra 
(Linchevski & Herscovics, 1996; Henningsen & Stein, 1997). Learners could then see the possible courses 
of action and choose those appropriate to the task or purpose (Kieran, 2004). Cognitive tutors addressed 
this problem by modelling all the known solution paths and prompting learners to choose one. Ill- 
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defined or unstructured domains of knowledge that relied on meaningful patterns or activities without a 
clear goal could not be modelled with ACT-based tutors (Mitrovic, 2012). 


Constraint-based intelligent tutors addressed knowledge domains beyond the reach of cognitive tutors. 
By modelling domains of knowledge with constraints, intelligent tutors focused on task constraints in a 
particular context. Any actions that fell outside the constraints that define the situation were flagged 
with a message identifying the constraint as violated and an explanation of the reason. For example, if a 
learner attempted to combine 3x + 1 as 4x, the system would respond with, "A linear term like 3x cannot 
be combined with a constant term like 1." The ability to identify patterns and their meanings made it 
possible to develop constraint-based tutors for fields like design (Mitrovic & Weerasinghe, 2009) and 
collaborative learning environments (Baghaei, Mitrovic, & Irwin, 2007). 


Despite advances in constraint-based modelling, intelligent algebra tutors, whether production rules- 
based or constraint-based, largely function the same way in learning situations. A student goes through 
a series of tasks on a computer or digital device guided by the automated tutor. Assessment items are 
either embedded in the learning sequence or given separately. Until now, constraint-based modelling 
has not been used as a reference for measurement of learning activity. 

3 METHODS 


This section begins with a description of the two learning environments and activities used in the study, 
followed by a description of data collection and the study participants. We then present a detailed 
description of the log file and constraint-referencing parsers that construct the cognitive complexity 
model used to measure task difficulty and trace individual performance over time. 

3.1 Classroom Network Activity Designs 

The study reported in this paper was conducted in the context of a larger ongoing design-based research 
project focused on developing and studying collaborative learning activities for middle- and high-school 
mathematics using classroom networking tools (White, 2006; 2013). Broadly, project designs aim to map 
the relationships among pairs or small groups of students to different aspects of a mathematical 
problem or situation through personal digital devices connected on a network. Posing tasks that require 
coordination of those elements helps to structure collaboration within the groups. Examples along these 
lines include 1) pairs of students each moving a single Cartesian point in order to jointly form linear 
functions (White, Wallace, & Lai, 2012), 2) groups of four students each manipulating different vertices 
in order to explore changing and invariant attributes of a shared quadrilateral (Lai & White, 2012; 2014), 
or 3) multiple students respectively analyzing one among multiple linked function representations in 
order to investigate a problem (White & Lai, 2008; White, 2009; White & Pea, 2011). 

All of these designs are intended to foster student participation in discussions and investigations of the 
mathematical relationships around which interactions between group members are organized. 
Consequently, prior research studies featuring these tools have primarily focused on the analysis of 
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discourse, interaction, and mathematical reasoning at the small group level. In contrast, the goal of the 
present study is to explore methodological tools and approaches that might complement group-level 
analyses of forms of interaction supported by these designs with automated and ongoing means of 
monitoring and examining the ways individual students reason and perform over time as they 
participate in these designed environments and collaborative learning activities. Moreover, and in 
contrast to other project designs situated in more open-ended and applied problem-solving contexts 
such as cryptography (White, 2009), the activities featured in the present study were specifically 
designed to be relevant in and responsive to classroom settings featuring more conventional curricular 
and instructional approaches and emphasizing procedural knowledge. 

The data for the present study were collected during implementation of two classroom network designs 
for an introductory Algebra course — one, called Terms and Operations, centred on polynomial 
expressions, and the other, called Two-Sides, focused on solving algebraic equations. Both designs 
utilized Texas Instruments graphing calculators and a Tl-Navigator™ classroom network in combination 
with the NetLogo modelling platform (Wilensky, 1999) and Hubnet architecture (Wilensky & Stroup, 
1999). To capture network activity data, the Terms and Operations activity environment (White, 
Sutherland, & Lai, 2010) was modified to capture extensive log file data of student inputs. The Two-Sides 
activity environment (Sutherland & White, 2011) was designed with log file capture in mind. Both 
activities are described in detail below. 


3.1.1 Terms and Operations 

Pairs of students construct a collective expression by adding, subtracting, multiplying, or dividing a series 
of monomial terms in the Terms and Operations activity environment. Each student uses the arrow keys 
on the calculator to control the position of an icon in a window on a shared display at the front of the 
classroom. The teacher populates the window with monomial terms that float around inside the same 


window. 



Figure 1: Terms and Operations Shared Display 


Each student can capture a term by moving the icon over a floating term. When a term has been 
successfully captured, the calculator prompts the student to add, subtract, multiply, or divide the term 
to combine it with the expression they are constructing. The calculator then prompts the student or her 
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partner to rewrite the resulting expression. Pairs of students either take turns capturing terms and 
rewriting expressions or one student captures a term and chooses an operation for the other student to 
rewrite. If the student successfully rewrites the expression with an equivalent form, the collective 
expression on the shared display is updated to reflect the changes the pair made. If the rewritten 
expression is not equivalent, the shared display does not update the changes and the expression 
remains as it was before the term was captured. 

a) b) 



Figure 2: Terms and Operations Student Calculator Screens. Student calculator screens featuring a) 
captured term and operation choices and b) a collective expression under the chosen term and 

operation and an equivalent student entry. 

The teachers conducted a variety of activities during data collection for this study. Four sessions focused 
on operating with integer terms. The teachers subsequently asked students to construct linear 
expressions, quadratic expressions, and expressions involving parentheses. Rewriting involved 
simplifying, factoring, distributing and combining like terms, and occasionally splitting terms. Students 
were often given the freedom to use any operation, but were sometimes required to use all four 
operations during expression construction or restrict themselves to various combinations of operations. 
Fifteen Terms and Operations sessions were included in this study. 

3.1.2 Two-Sides 

Each member of a student pair is assigned to either the left or right side of an algebraic equation in the 
Two-Sides activity environment. To create a new equation, each student in the pair must enter a valid 
algebraic expression. The networked relationship between the two students' calculators in the Two- 
Sides environment acts as an equal sign for the equation they share. Each member can replace the 
expression on her own side of the equation by entering any equivalent expression. When partners want 
to transform the entire equation, they must coordinate the transaction by pressing the operate button 
and entering equivalent operations on both sides. The system updates the equation only if the 
operations and transformations by both students are equivalent. Only transformations of the expression 
on one side of the equation were included in this study to align with the transformations recorded in the 
Terms and Operations environment. 
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Figure 3: Left-Side Student Entering an Equivalent Expression in Two-Sides. A student simplifies the 
left side of the equation after the pair has subtracted a 3 from both sides. The calculator display will 
update only if the student successfully enters an expression equivalent to (3x-l)-3. 


During Two-Sides sessions, students constructed linear and quadratic equations that were then solved 
by another pair. This study included two Two-Sides sessions. 


These two activity environments were developed as part of a larger research focus on technology- 
supported collaboration, which provided a context for the activities developed for this study. The 
designers endeavored to map social relations to mathematical constructs; in the first case by supporting 
joint construction of algebraic expressions; in the second by assigning opposite sides of an algebraic 
equation to each member of a dyad. This study focuses on the inputs of individual students captured 
from network activity to validate the constraint-referenced metrics of cognitive complexity. 
Collaborative focus of the activity designs supports subsequent analysis of student responses to periods 
of struggle. 

3.2 Data Collection and Participants 

Data collected in the spring semester of 2012 during classroom network activities was conducted in two 
algebra lab support classes (N=54 students) taught by the same teacher in a racially diverse 
comprehensive urban high school, grades 9-12, with 70% of the population of approximately 2000 
students identified as living in low socioeconomic conditions. All students included in the study were 
selected based upon concurrent enrollment in Algebra Lab and Algebra I classes. The block schedule at 
the high school meant that classes covered a full year course in Algebra in approximately 4.5 months, 
reported as two separate semester-long classes. Students were identified for placement in the Algebra 
Lab when they had been unable to pass the Algebra I course. 


During the sessions included in this study, 14,262 lines of log file data were collected from the classroom 
network environment. Each session was recorded with a wide-angle video of the classroom and a screen 
capture video of the shared display on the classroom network server. The data set includes 17 sessions 
(15 Terms and Operations; 2 Two-Sides) of algebra activities conducted on the classroom network. The 
parser identified 1794 transformations in which students attempted to replace an algebraic expression 
with an equivalent expression. 
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3.3 Log File Parser 

The log file parser filters out sequences of log file entries that do not involve mathematical 
transformations and aggregates each transformation sequence into a single line in canonical form. 
Transformation sequences differ for each activity environment, so the parser applies a different 
grammar to each. 


Table 1: Canonical Form of Transformation Sequences 

Description Type Example 

09:35:38 


Time stamp 

Time 

Group Number 

Integer 

Group Members 

String 

Code 

String 

Student taking action 

String 

Original Expression 

String 

Replacement Expression 

String 


Florence Ali 

Equivalent, Not Equivalent, Syntax Error 
Florence 
3X+5-X 
2X+5 


3.4 Constraint-Referencing Parser 

The constraint-referencing parser forms the core of the analytics engine developed for this study and 
contains a constraint-based model of the portion of algebra relevant to this study. On the first pass 
through the data, the parser grammar references the model to identify six elements of each algebraic 
expression. Algorithms in the analytics engine enumerate the elements and add additional fields to each 
record for the number of constant terms, linear terms, quadratic terms, negative symbols, parenthesis 
pairs, and a binary indicator to identify whether the parenthesis have mathematical meaning. 


Table 2: Expression Elements Identified by the Constraint-referencing Parser 


Element 

Description 

Constant Symbols 

How many constant symbols does the expression have? 

Example: 2x+3-l has 2 constant symbols. 

Linear Symbols 

How many linear symbols does the expression have? 

Example: 2x - 3 + x has 2 linear symbols. 

Quadratic Symbols 

How many quadratic symbols does the expression have? 

Example: 2x 2 +3x has 1 quadratic symbol. 

Negative Symbols 

How many negative symbols are contained in the expression? 

Example: —2(x—3) has 2 negative symbols. 

Parentheses Pairs 

How many parenthesis pairs are present in the expression? 

Example: 2x+(3-x) has 1 parenthesis pair. 

Nested? 

Would simply eliminating the parentheses change the mathematical value? 

Example: 2+(3x-3) the parentheses are not mathematically meaningful 
2(3x—3) the parentheses are mathematically meaningful 


Constraint-based models are able to represent ill-defined and unstructured domains of knowledge 
because they can assign semantic meaning to patterns (Muresan, 2013; Mitrovic, 2012). Referencing 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


151 








JOURNAL OF LEARNING ANALYTICS 


S8LAR 


(2016). Constraint-referenced analytics of algebra learning. Journal of Learning Analytics, 3(3), 143-169. 
http://dx.doi.Org/10.18608/jla.2016.33.8 

constraints in the model gives meaning to symbol patterns in the expression, just as students must 
recognize that a number next to an x is a linear term and must be kept separate from a constant term in 
certain contexts. Constraint referencing gives the analytics engine fine-grained access to the ways 
students struggle as they manipulate algebraic expressions. 


3.5 Model of Cognitive Complexity 

The analytics engine measures the cognitive complexity of transforming an algebraic expression by 
comparing the rate of success for replacing original expressions with a corresponding equivalent 
expression to a model of the cognitive complexity of the transformation. Each of the research streams 
discussed above suggests that the difficulty of making meaning in an algebraic context is related to the 
number of different algebraic elements that must be recognized and coordinated. To test this 
relationship, the constraint-referenced parser takes a second pass through the data to flag which of the 
six elements are present in the original expression and how they differ from the elements in the 
replacement expression. The complexity of the original expression defines the complexity of the 
algebraic context, whereas the differences in complexity between the original and replacement 
expressions define the complexity of the transformation attempt. The number of distinct kinds of 
elements the learner must coordinate is captured by the differences between the original and 
replacement expressions. 


Table 3: Record Fields After Second Constraint-referencing Parser Pass 


Description 

Type 

Example 

Time stamp 

Time 

09:35:38 

Group Number 

Integer 

2 

Group Members 

String 

Florence Ali 

Code 


String 

Equivalent, Not Equivalent, Syntax Error 

Student taking action 

String 

Florence 

Original Expression 

String 

3X+5-X 

Replacement Expression 

String 

2X+5 


Constant Terms 

Integer 

1 


Linear Terms 

Integer 

2 

cd 

Quadratic Terms 

Integer 

0 

a 

Negative Symbols 

Integer 

1 

L_ 

o 

Parenthesis Pairs 

Integer 

0 


Nested? 

Integer 

0 


Complexity 

Integer 

3 (Constant, Linear, Negative are present) 

-I—' 

c 

Constant Terms 

Integer 

i 

CD 

E 

Linear Terms 

Integer 

i 

CD 

U 

CD 

Quadratic Terms 

Integer 

0 

"q. 

CD 

Negative Symbols 

Integer 

0 

CC 

Parenthesis Pairs 

Integer 

0 


Nested? 

Integer 

0 


Complexity 

Integer 

2 (Constant and Linear are present) 

Transformation Complexity 

Integer 

1 (Difference between Original and Replacement) 
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Other models of complexity could be created that sum the total number of elements present, flag other 
operators (i.e., add, multiply, divide, square root, etc.), and take into the account the kinds of operations 
used during the transformation. Weights could be added to adjust the relative contributions each 
element makes to cognitive complexity. This simple model represents a first step toward testing the 
validity of constraint referencing as an approach to measuring cognitive difficulty and illustrating how 
the analytic results can shed light on learning activities. 

4 RESULTS 

In this section, we present psychometric validity of the cognitive model as an instrument for measuring 
task complexity and the resulting traces of individual student performance over time that reveal periods 
of struggle. 

4.1 Complexity Model Validation 

The constraint-referencing parser found 1795 attempts in both classes to make an equivalent 
transformation of an algebraic expression, 1394 successful and 401 unsuccessful. Table 6 shows the 
breakdown of attempts for both classes, and each class separately. Measurement against a constraint 
produces binary results — a transformation is either equivalent or it is not. 

Table 4: Equivalent Transformation Attempts for Both Classes 


Data Set 

Successful 

Unsuccessful 

Total 

Rate 

All Classes 

1394 

401 

1795 

.778 

Period 1 

732 

241 

973 

.752 

Period 3 

662 

160 

822 

.805 


Interval or continuous measurements can be taken against the constraints by establishing the 
appropriate scale. The complexity of the transformation constitutes an interval scale in the sense that 
each kind of element represents 1 point on the scale. The constraints in the model for algebra 
referenced by the parser define the meaning of each element. 

The binary nature of the dependent variable (equivalency of the transformation) indicates the use of 
logistic regression for statistical analysis. The logistic regression model characterizes the overall 
relationship between the independent and dependent variables. To account for repeated measures 
taken for each student and the potential influence of the differences between the two classes, student 
and class period were added as covariates. A plot of mean success rates (Figure 4a) has been added to 
show the influence of each level of complexity. The number of attempts students made at each level 
influences the interpretability of the mean as an indicator of the success at a particular level. 

For example, in the original expression complexity chart below, only one attempt was made at level 6. 
The level 6 attempt was successful. The 100% success rate at level 6 does not indicate that these 
students, or any other students in the class, will be equally successful given more attempts. As the 
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number of attempts increases, a truer picture of success rate comes into focus. We have included the 
histogram (Figure 5b) showing the number of attempts at each level to clarify the meaning of the plots. 

The cognitive complexity of the original expression had a moderate effect (eb=.603) but significant 
relationship to the success rate (x2=49.333, p<.001, df=l). The complexity of the transformation 
influenced the success rate (x2=165.990, p<.001, df=l) to a much greater degree (eb=.332). 




Cognitive Complexity of Original Expression 


Figure 4: Cognitive Complexity of the Original Expression: a) Equivalent Transformation Success Rate; 

b) Frequency of Attempts 




-10 12 3 4 


Cognitive Complexity of Transformation 


Figure 5: Complexity of the Transformation: a) Equivalent Transformation Success Rate; b) Frequency 

of Attempts 


Table 5 summarizes the results of logistic regression analysis. Transformations became increasingly 
more difficult as students attempted to make larger changes in expression complexity. The complexity 
of the transformation was associated with a much greater decline in success rate than the complexity of 
the original expression. 
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Table 5: Summary of Complexity Analysis for Both Classes. 


Complexity 

Log Odds 
(6) + 

Odds Ratio 
(e T 

Confidence Limits 
(e 6 at 95%) 

Waldx2 +++ 

(df=l) 

P 

Original 

-0.505 

0.603** 

0.524< e 6 <0.695 

49.333 

p<.001 

Transformation 

-1.102 

0.332** 

0.281< e 6 <0.393 

165.990 

p<.001 


SPSS 22™, IBM Corp, 2014 

+ Log Odds (6) in log units: Negative value indicates success rate decreased as complexity increased. 

++ Odds Ratio (e 6 ): Values less than 1 indicate that success rate decreased as complexity increased. 

+++ Wald x2: Reflects the deviance (variability) in the data. 

** Highly significant 

Notes: Student was included in the model as a repeated covariate and the influence was significant 

(p<.001). 

Class period was included as an indicator covariate but the influence was not significant. 

The Chi-squared statistic for transformations exceeded 150. This indicates that there was extensive 
deviance (variability) in the data and that the measurement instrument is sensitive to these differences. 
The high degree of significance indicates that the variability was highly systematic, even with students 
included as a repeated-measures covariate in the model. Taken together, these two statistics indicate 
that the cognitive analytics instrument measures the underlying construct and is highly discriminant, 
able to detect individual differences and individual trends overtime. 

4.2 Reliability Across Classes 

Constraint-referenced analytics detect the influence on the success rate of the cognitive complexity of 
the transformation attempts as we have defined them here. It also takes into account the complexities 
of the original expression. Students recorded very few (K=7, less than 1%) attempts to increase the 
complexity of the expression. By definition, the complexity of the original expression limits the potential 
complexity of simplification, because the amount of decrease cannot exceed the complexity of the 
original expression, and even in simplest terms, an expression has a complexity value greater than zero. 
Furthermore, while the influence of the complexity of the original expression on the success rate is 
significant, it does not account for much of the variability in the data. For these reasons, the class level 
analyses have been restricted to the complexity of the transformation attempts themselves, since they 
always imply a certain minimum level of complexity in the original expression. 

4.2.1 Class Comparisons 

The influence of complexity on success rate for each class followed a strikingly similar pattern to the 
overall results. Table 6 shows the means, magnitude, significance, and frequency for the influence of 
complexity on success rate at each level by class. The low frequencies highlighted in gray indicate that 
the results may be a poor indicator of the true mean at that level of complexity. The negative Beta 
values indicate that success rate decreased as the level of complexity in the transformation increased. 
The size of the exponentiated Beta (e e ) indicates the slope of the decrease — how much the success rate 
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decreased as the complexity increased. These results were highly significant and exerted a strong 
influence on the success rate for both the classes. 


Table 6: Class Means and Statistics for Cognitive Complexity 


Class 


-1 

0 

1 

2 

3 

4 

P 

e p 

P< 

Class A 

Means 

0.00 

0.91 

0.72 

0.53 

0.56 

0.25 

-1.07 

0.34* 

.001 

Attempts 

1 

348 

392 

174 

36 

4 


Means 

0.67 

0.91 

0.80 

0.52 

0.69 

0.33 




Class B 

Attempts 

3 

370 

312 

125 

16 

3 

-1.14 

0.32* 

.001 


*Significant at p<.001 


Figure 6 shows a chart of the mean success rates for both classes. The slightly higher success rate at 
complexity level 3 for Class B is most likely influenced by the difference in number of attempts for Class 
B (K=16) compared to Class A (K=36). The mean success rate reflects a truer picture as the number of 
data points increases. 



—©— Period 1 

♦ Period 3 


Figure 6: Class Success Rates by Complexity 


4.2.2 Tracking Individual Performance 

The results produced by the analytics engine can be used to trace the cognitive difficulty of sequences of 
transformation attempts — an automated form of repeated measures. By calculating a moving average 
of these repeated measures, a chart of performance over time can be produced. Table 7a shows success 
for each attempt and the complexity of each original expression and transformation. Table 7b illustrates 
how performance was computed as a moving average of success rate over five attempts for the first ten 
attempts made by Bryon. Moving averages for success rate, cognitive complexity of the original 
expression and the cognitive complexity of the transformation are found in Table 7c. Calculations for ten 
attempts produce six averages. Each successive range is moved one attempt to the right across the data 
set. The resulting values show how success rate changes over time (Figure 7a). 
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Table 7a: Complexity and Success for Individual Equivalent Transformation Attempts 


BRYON 

ATTl 

ATT2 

ATT3 

ATT4 

ATT5 

ATT6 

ATT7 

ATT8 

ATT9 

ATT10 

Success 

0 

1 

1 

1 

0 

1 

1 

0 

0 

0 

Original 

2 

2 

2 

2 

3 

3 

4 

3 

3 

3 

Transformation 

1 

1 

1 

1 

2 

2 

3 

2 

2 

1 


Table 7b: Moving Average as Success Rate Over a Range of 5 Attempts 

Success 

0 

1 

1 

1 

0 

1 

1 

0 

0 

0 

Success 

0 

1 

1 

1 

0 

1 

1 

0 

0 

0 

Success 

0 

1 

1 

1 

0 

1 

1 

0 

0 

0 

Success 

0 

1 

1 

1 

0 

1 

1 

0 

0 

0 

Success 

0 

1 

1 

1 

0 

1 

1 

0 

0 

0 

Success 

0 

1 

1 

1 

0 

1 

1 

0 

0 

0 

Moving Average 0.6 

0.8 

0.8 

0.6 

0.4 

0.4 






Table 7c: Moving Averages for Success, Original Expression, and Transformation Complexity 


BRYON 

Range 1 

Range 2 

Range 3 

Range 4 

Range 5 

Range 6 

Success 

0.6 

0.8 

0.8 

0.6 

0.4 

0.4 

Original 

2.2 

2.4 

2.8 

3 

3.2 

3.2 

Transformation 

1.2 

1.4 

1.8 

2 

2.2 

2 



Figure 7a: Success Rate Over 10 Attempts Using a Moving Average for Every 5 Attempts 


The moving averages for the cognitive complexity show how the original expression provided a context 
in which the transformation took place. The potential complexity of any attempt to simplify the original 
expression is limited by the complexity of the original expression. Bryon attempted to reduce the 
original expression to its simplest form in each case. The chart reveals that as the complexity of the 
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original expression increased, so did the amount of change Bryon attempted to make during the 
transformation. This upward trend in complexity was accompanied by a downward trend in success rate. 


Bryon Complexity 

6 


4 

2 

0 



Original 



■ Transformation 


R1 R2 R3 R4 R5 R6 


Figure 7b: Moving Average of Cognitive Complexity of Original Expression and Transformation 


4.2.3 Calculating Performance as the Product of Complexity and Success Rate 
Calculating performance as the product of transformation complexity and success rate reveals a clearer 
picture of Bryon's performance. The chart in Figure 8 represents the values found in Table 8. The green 
area of Figure 8 represents cognitive performance, the proportion of transformation complexity for 
which Bryon made successful transformations. The exposed red portion shows the performance 
difference between expected performance and actual performance. A wide red area indicates a period 
of struggle when the student experienced a low success rate at a certain level of complexity. In this case, 
as the complexity level increased, performance decreased and the difference between actual and 
expected performance (the exposed red area) widened. Calculating success rate as a proportion of 
transformation complexity made it possible to chart performance over time. 


Table 8: Performance Calculated by Success Rate as a Proportion of Transformation 

Complexity 


BRYON 

Range 1 

Range 2 

Range 3 

Range 4 

Range 5 

Range 6 

Success 

0.6 

0.8 

0.8 

0.6 

0.4 

0.4 

Transformation 

1.2 

1.4 

1.8 

2 

2.2 

2 

Cognitive Performance 

0.7 

1.1 

1.4 

1.2 

0.9 

0.8 


5 

4 

3 

2 

1 

0 


Bryon Performance 



Original 

Complexity 

■ Transformatio 
n Complexity 


R1 R2 R3 R4 R5 R6 
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Figure 8: Performance Over Time 

4.2.4 Analysis of Performance Charts 
Examination of performance charts for 49 students in Class A and Class B revealed that students 
experienced 92 periods of struggle and responded in three distinct ways: 1) they persisted in attempting 
similar transformations until they experienced a moment of insight that led to subsequent success; 2) 
they abandoned the struggle and switched to familiar transformations; or 3) they stopped participating 
either by entering no more inputs though they attended class, or depended entirely upon the partner to 
dictate inputs. 

4.2.5 Persistence That Led to a Conceptual Shift 

Michiko (Figure 9) experienced a period of struggle and was able to improve cognitive performance and 
gradually eliminate the performance difference by improving her success rate. She attempted 13 
consecutive transformations with an average cognitive complexity of approximately 1. She struggled 
initially as indicated by the wide performance difference from range R27 through R30 (eight attempts). 
She narrowed the performance difference by becoming more successful while maintaining a consistent 
level of cognitive complexity. The result was an increase in cognitive performance of nearly 1 complexity 
unit, eliminating the difference between expected performance (Transformation Complexity) and actual 
performance. 


S8LAR 


MICHIKO 

Cognitive Performance 

Original 

Complexity 

■ Transformation 
Complexity 

Cognitive 

Performance 


Figure 9: Improved Performance at Constant Level of Complexity 

A look at the detailed record of attempts during that sequence reveals that Michiko gained 
understanding of the underlying principle that linear terms and constant terms must remain separate 
when combining like terms. 



R27 R28 R29 R30 R31 R32 R33 R34 
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Table 9: Sequence of Transformations Associated with Learning for Michiko 


Attempt 

Partner 

Original 

Replacement 

Success? 

27 

Erik 

-2X+1 

-IX 

No 

28 

Erik 

-2X+4X 

2X 

Yes 

29 

Erik 

2X+1 

3X 

No 

30 

Erik 

3X-1-7X 

10X-1 

No 

31 

Erik 

3X-1+-5 

3X+6 

No 

32 

Tabatha 

4X-1 

3X 

No 

33 

Tabatha 

0+7X 

7X 

Yes 

34 

Tabatha 

7X+2 

9X 

No 

35 

Tabatha 

7X+2X 

9X 

Yes 

36 

Tabatha 

9X+X 

10X 

Yes 

37 

Tabatha 

(10X+-7)—(-7) 

10X 

Yes 

38 

Tabatha 

10X—(—6) 

10X+6 

Yes 

39 

Tabatha 

10X+6+-6 

10X+0 

Yes 

40 

Tabatha 

10X+0+-7 

10X+7 

No 

41 

Tabatha 

10X+0+-5 

10X+5 

No 

42 

Tabatha 

10X+0+-5 

10X-5 

Yes 

43 

Tabatha 

10X-5+-5 

10X+0 

No 


All the transformations attempted by Michiko during this sequence involved combining like terms in 
linear expressions. Michiko mistakenly combined linear and constant terms (27-37), successfully making 
transformations only when the terms in the original expression were already alike (28, 35, and 36) or the 
constant term was a zero (33). A shift in understanding occurred beginning with attempt 37. Michiko 
replaced (10x+-7)-(-7) with lOx, recognizing the underlying principle that the linear term lOx and the 
constant terms -7 and -7 must be kept separate. Attempts 38 and 39 confirm that Michiko recognized 
unlike terms and combined them appropriately. Attempts 40 through 43 contain mistakes related to 
negative values, but linear and constant terms are not confused again in this sequence. 

Michiko worked with two different partners. She made only one successful attempt when partnered 
with Erik (28), combining terms that were already alike in the original expression. She experienced 
success for the same reason when paired with Tabatha (35 and 36). The insight that led to the ability to 
differentiate linear and constant terms occurred when Michiko was paired with Tabatha. To what 
degree this shift in partnership led to new understanding is not clear from this record, since the same 
mistakes took place with both partners. 

Only ten periods of struggle that led to better understanding of a principle underlying a procedure were 
found in this study. All periods of struggle that led to learning could be uniquely identified by the same 
five conditions. 
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Span 

Complexity 
Similarity 
Performance 
Difference 
Conceptual Shift 


Table 10: Conditions for Learning 

Period of struggle spanned at least 7 attempts. 

The transformation complexity remained relatively constant. 

The student attempted similar transformations and made similar mistakes. 

A performance difference of at least 60% was eliminated during the sequence. 

The student experienced a conceptual shift and was subsequently successful. 


4.2.6 Abandoning the Struggle 

Marcel experienced a mild period of struggle prior to the extended period of success. The level of 
complexity decreased until the performance difference disappeared (Figure 10). A review of the record 
reveals that Marcel struggled with combining quadratic and linear terms and quickly switched to more 
familiar transformations. 



Rll 


R12 


R13 


R14 


MARCEL 
Cognitive Peformance 

Original 
Complexity 

■ Transformation 
Complexity 


R15 


Cognitive 

Performance 


Figure 10: Period of Struggle Abandoned 


Table 11: Marcel Avoided Quadratic Expressions 


Attempt 

Partner 

Original 

Replacement 

Success? 

11 

Darryl 

0+2X 2 

2X 2 

Yes 

12 

Darryl 

2X 2 +1 

3X 2 

No 

13 

Darryl 

0+2X 2 

2X 2 

Yes 

14 

Darryl 

2X 2 +4X 2 

6X 3 

No 

15 

Darryl 

0+2X 

2X 

Yes 

16 

Darryl 

2X+7X 

9X 

Yes 

17 

Darryl 

10X-10+-3 

10X-13 

Yes 

18 

Darryl 

10X-20+-5 

1OX-25 

Yes 

19 

Darryl 

(-3XH-1) 

3X 

Yes 


Marcel made four consecutive attempts to transform an expression with a quadratic binomial 
expression, experiencing success only when the constant term was zero (Attempts 11 and 13). To 
resolve the struggle, he returned to familiar linear expressions. He mistakenly replaced 2x 2 +l with 3x 2 
(Attempt 12) and inappropriately incremented the exponent when he replaced 2x 2 +4x 2 with 6x 3 
(Attempt 14). The struggle ended when he switched to linear expressions with consistent success. 
Marcel went on to make 15 consecutive successful transformations with linear expressions. Only two 
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additional attempts to transform quadratic expressions were found in the record after this sequence 
one was successful and the other unsuccessful. Resolving the struggle meant avoiding the quadratic 
transformations that Marcel found difficult. Most periods of struggle followed the same pattern as 
Marcel (K=76). 

4.2.7 Non-Participation 

A few students chose non-participation in response to periods of struggle (K=6). Four students 
attempted equivalent transformations during one or two sessions and subsequently ceased to 
participate at all though records indicate they attended class on the days the sessions were conducted. 

Two students continued to input expressions into the calculators over several sessions. However, 
records show that increases and decreases in performance, as well as the kinds of transformations that 
could be successfully completed aligned with different partnerships. The case of Jessie illustrates this 
kind of non-participation by partner proxy. 


JESSIE 

Cognitive Performance 

Original 

Complexity 


■ Transformation 
Complexity 


Cognitive 

R7 Performance 


Figure 11: Non-participation by Partner Proxy 

Jessie appears to narrow the performance difference by reducing the complexity of the transformations 
he attempted. The record shows that his apparent success can be explained by changes in partnership. 


Table 12: Changes in Partnership Account for Success Rate 


Attempt 

Partner 

Original 

Replacement 

Success? 

1 

Arlene 

0+-7X 

0 

No 

2 

Arlene 

0-X+-5X 

0 

No 

3 

Arlene 

0-X+-6X+7X 

-14X 

No 

4 

Arlene 

0—(—4) 

0 

No 

5 

Arlene 

0-(-4)+4 

4 

No 

6 

Arlene 

0—(—4)+7 

45552 

No 

7 

Bryon 

X A 2+X+3+2X+5X+-5X 

X A 2+3X+3 

Yes 

8 

Jeffry 

7+7 

14 

Yes 

9 

Jeffry 

14*2 

28 

Yes 

10 

Jeffry 

4*7 

28 

Yes 
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The chart for Jessie (Figure 11) reveals a modest increase in performance that might be associated with 
learning during an intense period of struggle. He was unable to complete any successful transformations 
in the first six attempts. When he switched partners and joined Bryon (Attempt 7), he was able to 
reduce a quadratic expression to simplest terms successfully. When he joined Jeffry, he was able to 
handle integer expressions (Attempts 8-10). Throughout the 12 sessions in which Jessie participated, 
success rate, complexity, and type of transformation were dependent upon the partnership. Jessie 
chose non-participation through partner proxy. This pattern was found for only one other participant. 

Table 13: Summary of Performance Chart Analysis 


Response 

Count 

Learning 

10 

Challenge Reduction 

76 

Non-participation 

6 

Total 

92 


5 DISCUSSION 

Constraint-referenced analytics of algebra learning activities came from the two-fold desire to take a 
quantitative look at student responses in intentionally collaborative algebra activities, and to situate 
those activities in a more traditional introductory algebra classroom setting focusing on procedural 
understanding. Three constraint-based models were developed to measure the complexity of attempts 
to transform an algebraic expression by replacing it with an equivalent expression. The cognitive 
complexity model produced psychometrically verifiable results, showing decreased performance as 
complexity increased. The instrument proved to be highly discriminant, making an appropriate tool for 
measuring individual performance. The results were reliable across two classes of students with the 
same teacher. Monitoring student inputs over time revealed periods of struggle and three distinct 
patterns of student response to periods of struggle. Students who persisted by attempting similar 
transformations were eventually successful and were able to complete similar transformations 
afterwards. The majority of the students abandoned the transformations they found difficult and 
returned to more familiar algebraic forms. A few students chose to stop participating either by allowing 
their partner to guide their attempts or by ceasing to make any attempts at all. These patterns of 
response to struggle were consistent across two classes taught by the same teacher. 

Students who persisted by repeatedly attempting similar transformations experienced an apparent 
conceptual shift in understanding. The case of Michiko illustrates this pattern of response as she 
attempted similar transformations until a moment of insight led to an apparent shift in understanding 
the underlying principle that linear and constant terms are different classes of mathematical objects and 
cannot be combined for a transformation to be equivalent. The other nine instances of learning were 
each accompanied by a similar apparent shift in understanding of a particular principle underlying the 
procedure being attempted. Whether these apparent conceptual shifts were associated with being 
taught and a matter of rote learning (Kapur, 2011), or whether they were discovered by the student was 
not observed. Additional independent evidence would be necessary to confirm the nature and depth of 
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a change in conceptual understanding. The difference between explicit and implicit conceptual 
knowledge is tightly tied to the task demands (Crooks & Alibali, 2014). Students were not asked to 
explain their reasoning, so any suggestion of conceptual knowledge change can only be implicit in 
nature. 


The students who participated in this study had been unsuccessful in a previous algebra class and were 
taking two periods of algebra simultaneously. It is not clear whether the heavy load of algebra classes 
contributed to the non-participation and challenge reduction rates reported here. It is possible that 
students enrolled in only one algebra class at a time would be more likely to persist during periods of 
struggle. The collaborative activity designs make it difficult for students to work alone. The study does 
not address the possibility that student willingness to persist was influenced by the partnership between 
students. It is also possible that student inputs were influenced by instruction from the teacher or by 
observation of other groups, making it prudent to take these results as a finding for the usefulness of 
the constraint-referenced analytics technique and a very preliminary description of the ways students 
respond to periods of struggle. 

6 CONCLUSION 

Despite significant limitations, the constraint-referencing engine designed for this study produced 
psychometrically verifiable results that made possible the tracking of student activity over time. Analysis 
did reveal periods of struggle and uniquely identified sequences in which a conceptual shift appeared to 
take place, consistent with a definition of conceptual understanding as knowledge of principles 
underlying procedures (Crooks & Alibali, 2014). All cases of apparent learning featured a shift from 
struggle to success that may signal a conceptual shift. The analysis for this study took place post hoc. 
However, constraint-referenced analytics does not interrupt the learning process, making it possible to 
monitor algebraic transformation activities in real time. By making teachers aware when students are 
struggling, and by identifying patterns of response to the struggle, the system may become useful to 
support learning. The constraint-referenced analytics methods described here are in their earliest stages 
of development. The promising results presented here should be viewed as preliminary and exploratory. 

Constraint-based models do define domains of knowledge based upon contextual patterns and are 
capable of capturing and encoding semantic meaning, but this study only makes use of this approach at 
the most fine-grained elemental level by encoding basic term types, negative symbols, and parentheses 
pairs. More full-featured grammars and pattern-matching technology can model complex and ill-defined 
domains of knowledge. Mathematical models can be developed to address multiple representations, 
operations, and patterns. Referencing constraints provides analytics developers a binary measure of 
success when actions fall inside or outside constraints. Models of difficulty can be developed based 
upon the number and kinds of constraints in play. Schemes for characterizing the relative influence of 
certain constraints over others promise to make models of complexity available for numerous domains 
of knowledge. 
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